Shore Side Data System (600031 and 900823) : Ingest Architecture
This page last changed on Mar 18, 2011 by kgomes.
This doc has been moved

Most of the documentation for SSDS should be on the Google code site now and this page is here:
http://code.google.com/p/shore-side-data-system/wiki/TransmogrifyAndIngest

Name Type Description
StreamID java.lang.short This basically states that the bytes are coming from a SIAM ExportablePacket class. SIAM uses constants defined in the org.mbari.siam.distributed.Exportable.java class to enumerate things like this and the short value for this is always 0x0100. SSDS Doesn't really care so we essentially ignore it.
DevicePacketVersion java.lang.long This is the "serialVersionUID" on the class that was used to export the bytes. It is the version of SIAM class that generated the byte array. SSDS Does not really care and as of this writing, it is always 0.
SourceID java.lang.long The ID of the device that the message was generated by.
Timestamp java.lang.long Epoch milliseconds (number of elapsed milliseconds since 1/1/1970 00:00:00) that the packet was generated by the device
SequenceNumber java.lang.long A number that is supposed to show the order of generation of packets from the device
MetadataRef java.lang.long This is the sequence number of the packet that contains the metadata that describes the information in this packet
ParentID java.lang.long The ID of the device that the generated device was attached to when it generated the packet
RecordType java.lang.long The type of record that this packet contains:
  • 0 = MetadataPacket
  • 1 = Non-MetadataPacket (Data and other)
SecondStreamID java.lang.short This defines the type of DevicePacket that was used to construct the byte array. The values are as follows:
  1. MetadataPacket = 0x101
  2. SensorDataPacket = 0x102
  3. DeviceMessagePacket = 0x103
  4. SummaryPacket = 0x102 (same as SensorDataPacket)
SecondPacketVersion java.lang.long This is the "serialVersionUID" on the class that was used to export the bytes. It is the version of SIAM class that generated the byte array. As of this writing, it is the same as the DevicePacketVersion. Since currently it is always 0, SSDS ignores it.
FirstBufferLength java.lang.int This is the length of the array that holds the bytes of the first buffer
FirstBuffer java.lang.byte [] This is the bytes array that represents the first buffer
SecondBufferLength java.lang.int This is the length of the array that holds the bytes of the second buffer.
SecondBuffer java.lang.byte [] This is the array that holds the bytes of the second buffer.

Now, in order to handle both types of inputs in Transmogrify (DevicePackets and BytesMessage structure), Transmogrify would take both and convert to a common format that would contain the information to cover both types of messages. Since the BytesMessage structure encompasses all the information in the DevicePacket, we simply used that byte structure and in Transmogrify, a DevicePacket is converted to a SSDSDevicePacket which is then converted to the same BytesMessage structure using the SSDSDevicePacket.convertToPublishableByteArray method. So at the end of the Transmogrify process, we have on byte array that is in the form of the diagram above that will then be used to publish a message to the next component which is Ingest. Transmogrify takes the SIAM byte array structure and converts it to the SSDS native byte array structure:

Notes on the conversion:

  1. The DevicePacketVersion, SecondStreamID, and SecondPacketVersion are used to determine the correct packetType (although, right now, DevicePacketVersion and SecondPacketVersion are ignored).
    • For MetadataPackets, the packetType is 1.
    • For SensorDataPackets, the packetType is 0 (SummaryPackets come across as SensorDataPacket and are differentiated by their recordType).
    • For DeviceMessagePackets, the packetType is 4.
  2. If the incoming packet is a MetadataPacket, the packetSubType is set to 0. Otherwise, it is set to the RecordType field.
  3. The RecordType is set to zero if the packet is a MetadataPacket and set equal to the RecordType from SIAM if not a MetadataPacket.
  4. The MetadataSequenceNumber is calculated depending on the device, it's parent, and the XML that is in it's payload. There is a component called the SIAMMetadataTracker that keeps track of this information and looks for real XML changes which is what should fire a change in metadata.
  5. The buffers are swapped if it is a MetadataPacket. It always seemed to logical to do it that way.
  6. This timestamp (epoch milliseconds) is split into seconds and nanoseconds.
Message Size Limitation!

Please note that because byte arrays are limited to 32 bit sizes, the largest payload of a message that can be converted by SSDS is 2GB. While this does not seem like a major restriction, it can be hit if somebody is using straight JMS messaging (or other) and makes a payload bigger than 2GB. SSDS will just ignore such a message.

Ingest Packet Structure

So now we have all messages coming into Ingest in a format that SSDS is expecting (i.e. that matches the SSDS view of the world). For the diagram in the previous section, the attributes in the SSDS Bytes Array are:

Attribute Type Description
sourceID java.lang.long This is what is known as the SSDS ID for the device (i.e. DeviceID) that actually generated the packet of information.
parentID java.lang.long This is the SSDS ID for the parent that the device was connected to when it sent the packet. If the ID is zero (0), then the generating device was not connected to a parent.
packetType java.lang.int This is the "Type" of packet that is being sent. It is basically an enumerated list the with following context:
0 = Data Packet
1 = Metadata Packet
2 =
3 =
4 = Device Message Packet
packetSubType java.lang.long This is the equivalent of the "recordType" listed in the Transmogrify component. It is used to provide the hook to tell the client applications what type of record this is that was sent. It really only has meaning in the context of data packets as a device can often send data packets of different formats. This basically tells the application which record form is being sent in this packet.
metadataSequenceNumber java.lang.long Also referred to as dataDescriptionID
dataDescriptionVersion java.lang.long  
timestampSeconds java.lang.long)  
timestampNanoseconds java.lang.long  
sequenceNumber java.lang.long)  
bufferLen java.lang.int  
bufferBytes java.lang.byte[bufferLen]  
bufferTwoLen java.lang.int  
bufferTwoBytes java.lang.byte[bufferTwoLen]  

The Ingest Message Driven Bean (MDB) then takes that byte array and using a PacketOutput class that corresponds to the correct source ID, metadataSequenceNumber, packetSubType, and parentID, it writes the packet to disk. It then uses a PacketSQLOutput to write that same packet to a table in the database.

Packet Translations

So, through all this, there are basically four representations of data packets in the SSDS ecosystem:

  1. SIAM Device Packet (and its sub classes MetadataPacket, SensorDataPacket, DeviceMessagePacket)
  2. SSDSDevicePacket (and its sub class SSDSGeoLocatedDevicePacket)
  3. SIAM Byte array (from Exportable class)
  4. SSDS Byte array

Here is a diagram of these various forms of data

And the translation rules (some of these may seem very strange for legacy reasons).

DevicePacket to SSDSDevicePacket

This translation is done in the constructor of SSDSDevicePacket which takes in a DevicePacket

DevicePacket Translation Rule SSDSDevicePacket
sourceID direct copy sourceID
systemTime direct copy systemTime
sequenceNo direct copy sequenceNo
metadataRef direct copy:
  1. metadataRef->metadataRef
  2. metadataRef->metadataSequenceNumber
  3. metadataRef->dataDescriptionID
  1. metadataRef
  2. metadataSequenceNumber
  3. dataDescriptionID
parentId direct copy:
  1. parentId->parentId
  2. parentId->platformID
  1. parentId
  2. platformID
recordType
  1. MetadataPacket: recordType to 0
  2. Other: direct copy
 recordType
 
  1. If MetadataPacket, packetType = 0
  2. If SensorDataPacket, packetType = 1
  3. If DeviceMessagePacket, packetType = 2
 packetType
firstBufferLength ignored  
firstBuffer First buffer depends on which type of packet
  1. If MetadataPacket, copy "bytes" buffer
  2. If SensorDataPacket, copy "dataBuffer"
  3. If DeviceMessagePacket, copy "message"
 firstBuffer
secondBufferLength ignored  
secondBuffer Only exists if MetadataPacket and will copy over "cause" buffer secondBuffer
DevicePacket to SIAM Byte Array

This is done by the SIAM Exportable Packet class

DevicePacket Translation Rule SIAM Byte Array
  This is a static value that is set to indicate the byte array is a DevicePacket and is set to 0x0100 EX_DEVICEPACKET
  This is just the serial version UID of the class which is always 0 serialVersionUID
sourceID direct copy sourceID
systemTime direct copy systemTime
sequenceNo direct copy sequenceNo
metadataRef direct copy metadataRef
parentId direct copy parentId
recordType direct copy recordType
  This is set based on what type of packet:
  1. If MetadataPacket, set to 0x0101
  2. If SensorDataPacket, set to 0x0102
  3. If DeviceMessagePacket, set to 0x0103
 EX_XXXXXXPACKET
  This is just the serial version UID of the class which is always 0 serialVersionUID
firstBufferLength direct copy (see note for first buffer) firstBufferLength
firstBuffer This depends on the type of packet:
  1. If MetadataPacket, "cause" bytes are copied over
  2. If SensorDataPacket, "dataMessage" bytes are copied over
  3. If DeviceMessagePacket, "message" bytes are copied over
 firstBuffer
secondBufferLength only valid with MetadataPacket, but is copied directly over secondBufferLength
secondBuffer only valid with MetadataPacket, and the "buffer" bytes are copied over secondBuffer
SSDSDevicePacket to SIAM Byte Array

Originally done in TransmogrifyMDB by calling SSDSDevicePacket.convertToPublishableVersion3ByteArray before passing byte array to method to translate to SSDS format and then send to ingest

SSDSDevicePacket Translation Rule SIAM Byte Array
  This is a static value that is set to indicate the byte array is a DevicePacket and is set to 0x0100 EX_DEVICEPACKET
  This is just the serial version UID of the class which is always 0 serialVersionUID
sourceID direct copy sourceID
systemTime direct copy systemTime
sequenceNo direct copy sequenceNo
metadataSequenceNumber direct copy metadataRef
parentID direct copy parentID
recordType
  1. If packetType = 0, set recordType = 0
  2. If packetType = 1, set recordType = recordType
  3. If packetType = 2, set recordType = recordType
 recordType
  This is set based on what type of packet:
  1. If packetType = 0, set to 0x0101
  2. If packetType = 1, set to 0x0102
  3. If packetType = 2, set to 0x0103
 EX_XXXXXXPACKET
  This is just the serial version UID of the class which is always 0 serialVersionUID
  This depends on the type of packet:
  1. If packetType = 0, set to length of "otherBuffer"
  2. If packetType = 1, set to length of "dataBuffer"
  3. If packetType = 2, set to length of "dataBuffer"
 firstBufferLength
other/dataBuffer This depends on the type of packet:
  1. If packetType = 0, set to bytes from "otherBuffer"
  2. If packetType = 1, set to bytes from "dataBuffer"
  3. If packetType = 2, set to bytes from "dataBuffer"
 firstBuffer
  This only exists if it is packetType = 0 then it is set to the length of the "dataBuffer" secondBufferLength
dataBuffer This only exists if it is packetType = 0 then it is set to the byte from the "dataBuffer" secondBuffer
SSDSDevicePacket to SSDS Byte Array

Originally done in SSDSDevicePacket.convertToVersion3ByteArray

SSDSDevicePacket Translation Rule SSDS Byte Array
sourceID direct copy sourceID
systemTime ignored during the translation directly, but used through getter methods for seconds and nanoseconds  
timestampSeconds direct copy (note that this is sort of a direct copy, there are getter methods on SSDSDevicePacket that convert the systemTime to seconds and nanoseconds when called). timestampSeconds
timestampNanoseconds direct copy (note that this is sort of a direct copy, there are getter methods on SSDSDevicePacket that convert the systemTime to seconds and nanoseconds when called). timestampNanoseconds
sequenceNo direct copy sequenceNumber
metadataRef ignored  
parentID ignored  
recordType If packetType = 0, set packetSubType to 0, otherwise set to recordType packetSubType
packetType Depends on packetType:
  1. If packetType = 0, set to 1
  2. If packetType = 1, set to 0
  3. If packetType = 2, set to 4
 packetType
metadataSequenceNumber direct copy metadataSequenceNumber
dataDescriptionVersion direct copy dataDescriptionVersion
platformID direct copy parentID
  copy length of dataBuffer firstBufferLength
dataBuffer direct copy firstBuffer
  copy length of otherBuffer secondBufferLength
otherBuffer direct copy secondBuffer
SIAM Byte Array to SSDS Byte Array

Originally done in TransmogrifyMDB in checkAndPublishBytes method

SIAM Byte Array Translation Rules SSDS Byte Array
EX_DEVICEPACKET ignored  
serialVersionUID ignored  
sourceID direct copy sourceID
  Depending on EX_XXXXXXXPACKET:
  1. If MetadataPacket, set packetType to 1
  2. If SensorDataPacket, set packetType to 0
  3. If DeviceMessagePacket, set packetType to 4
 packetType
  This was set using the SIAMMetadataTracker that tried to keep track of real version numbers based on XML in payload metadataSequencNumber
systemTime Split into timestampSeconds and timestampNanoseconds
  1. timestampSeconds
  2. timestampNanoSeconds
sequenceNo direct copy sequenceNumber
metadataRef direct copy dataDescriptionVersion
parentId direct copy parentID
recordType If MetadataPacket (determined from EX_XXXXXXXPACKET), recordType set to 0, otherwise set to recordType packetSubType
EX_XXXXXXXPACKET ignored in storage, but used in logic  
serialVersionUID ignored  
first/secondBufferLength If MetadataPacket (determined from EX_XXXXXXXPACKET), firstBufferLength is set to secondBufferLength so we can flip the "cause" and "buffer" bytes because it just made more sense since the cause was rarely populated. Otherwise set to firstBufferLength firstBufferLength
first/secondBuffer If MetadataPacket (determined from EX_XXXXXXXPACKET), firstBuffer is set to secondBuffer so we can flip the "cause" and "buffer" bytes because it just made more sense since the cause was rarely populated. Otherwise set to firstBuffer firstBuffer
firstBufferLength If MetadataPacket (determined from EX_XXXXXXXPACKET), secondBufferLength is set to firstBufferLength to flip "cause" and "buffer" bytes secondBufferLength
firstBuffer If MetadataPacket (determined from EX_XXXXXXXPACKET), secondBuffer is set to firstBuffer to flip "cause" and "buffer" bytes secondBuffer
Exploration of Upgrade of Ingest/Transmogrify to AMQP

In an effort to allow non-Java clients to send data to SSDS in the form of messages and to upgrade the messaging system to a technology that is more scalable and higher performance, an investigation of AMQP implementations was done.

  1. Qpid Exploration

Attachments:
SSDS JMS (text/xml)
SSDS JMS (text/xml)
SSDS JMS (text/xml)
SSDS JMS (text/xml)
SSDS JMS (text/xml)
SSDS JMS (text/xml)
SSDS JMS (text/xml)
SSDS JMS (text/xml)
SSDS JMS (text/xml)
SSDS JMS (text/xml)
SSDS JMS (text/xml)
SSDS JMS (text/xml)
SSDS JMS (text/xml)
SSDS JMS (text/xml)
SSDS JMS (text/xml)
SSDS JMS (text/xml)
SSDS JMS (text/xml)
SSDS JMS (text/xml)
SSDS JMS (text/xml)
SIAM Externalizable Structure (text/xml)
SIAM Externalizable Structure (text/xml)
SIAM Externalizable Structure (text/xml)
SIAM Externalizable Structure (text/xml)
SIAM Externalizable Structure (text/xml)
SIAM Externalizable Structure (text/xml)
SIAM Externalizable Structure (text/xml)
Device Packets.jpg (image/jpeg)
Device Packets.jpg (image/jpeg)
Device Packets.jpg (image/jpeg)
SSDSByteArrayFormat (text/xml)
SSDSByteArrayFormat (text/xml)
SSDSByteArrayFormat (text/xml)
SSDSByteArrayFormat (text/xml)
SSDSByteArrayFormat (text/xml)
SSDSByteArrayFormat (text/xml)
SSDSByteArrayFormat (text/xml)
SSDSByteArrayFormat (text/xml)
SSDSByteArrayFormat (text/xml)
SSDSByteArrayFormat (text/xml)
SSDSByteArrayFormat (text/xml)
SSDSByteArrayFormat (text/xml)
Transmogrify Steps.jpg (image/jpeg)
Document generated by Confluence on Feb 23, 2017 15:08